Batch Evaluation Metrics in Information Retrieval: Measures, Scales, and Meaning

A sequence of recent papers, including in this journal, has considered the role measurement scales information retrieval (IR) experimentation, and presented argument that (only) uniform-step interval should be used. Hence, it been argued, well-known metrics such as reciprocal rank, expected normalized discounted cumulative gain, average precision, either discarded tools, or adapted so their metric values lie at uniformly-spaced points on number line. These papers paint a rather bleak picture past decades IR evaluation, odds with community’s overall emphasis practical experimentation measurable improvement. Our purpose work is to challenge pessimistic assessment. In particular, we argue mappings from categorical ordinal data sets line are valid provided there an external reason for each target point have selected. We first consider general scales, categorical, ordinal, interval, ratio, absolute collections. connection two those categories also provide examples knowledge captured represented by numeric real Focusing then retrieval, document rankings data, effectiveness single value summarizes usefulness user population users any given ranking, able continuous variable ratio scale. That is, most current well-founded, and, moreover, more meaningful form than proposed “intervalized” versions.